Using NER with Index Entities (DRUID versions prior to 5.15)

Index entities are entities, which are trained for Natural Language Processing. Use them when you want to validate user input with a list of values predefined for your specific situation.

This section describes how to use NER with index entities in DRUID versions prior to 5.15.

Prerequisite

• You must have at least one entity defined in the system used by the bot. The entity you want to index should have a flat data type field (a field of type other than Entity or Entity List).

To use NER with Index Entities, follow these steps:

Step 1. Link your entity with an external data source

Connect to an external data source. Create a connector action that brings the values from an external data source and stores them in your entity.

When creating the connector action, make sure to select your entity as both the Request and Response Entity and select the Is Response Collection option.

As you are using the integration tasks only to read data from the external data source, do not provide any input arguments on the request. In the response, map the field from the data source to the DRUID entity field. You will index your entity by the field you map here.

For more information on how to connect to external data sources, see Integrations.

Step 2: Index the entity

2.1. Define bot-entity relation

Select the bot and in Bot Designer click the Entities tab. Initially, the Entities page is empty.

The Entities page will load all entities used by the respective bot (any entity which is related to another bot-related element, like flows, connector actions, etc). In case an additional entity relation needs to be added (an entity which is not yet used in any other bot-related element), the Add Bot Entity Relation button can be used to add a new empty line in the table.

Using inline editing, select the Entity, Matching Fields, Training Sources and tick the Is NLP check box. The table below describes the parameters for setting up an entity for NLE.

Parameter	Description
Entity	Select the entity you want to index.
Matching Fields	Select the field you used to map the data from your external data to DRUID entity field. Note:  For DRUID version 1.74 and higher, if you're using an entity on multilanguage bots, make sure to select the main field that has language specific field clones (E.g., index the entity [[Account]] by the field Name, where the field [[Account]].Name has the option Holds multilanguage records option enabled.
Training Sources	Select the connector action that will pull the selected field values from your external data source (i.e., database, web service, etc.).
Is NLP	Tick the check box if the entity will be used by NLP as well.

To save the entity, click the Save icon displayed inline with the record. The fields become non-editable and the Edit Schedule () icon displays in the Actions column in line with the entity.

The bot-entity relation is created in the system, but the entity is not trained for NLP.

Now you have to train NER for the entity, that is creating an entity index.

2.2. Create entity index

When you train the entity, DRUID calls the connector action which brings from the data source the list of values which needs to be indexed and saves them in a special DRUID index associated to your entity.

To manually create an entity index, click the Edit Schedule icon. A window appears. Click the Train button.

An entity index is created by the system for the selected entity.

If the external data source updates periodically, we recommend you to schedule the entity index training.

To schedule the training of the entity index, follow these steps:

1. Click the Edit Schedule icon displayed inline with your entity. A window appears.
2. Click the Settings section header. The Settings area expands.
3. Tick the Is scheduler enabled check box.
4. (Optional) Set specific NER parameters for the selected entity. For more information, see Entity NER-specific Parameters.
5. Set Start date and End date or tick No end date if you do not want to set an end date.
6. Select the Frequency (to run on specific day(s) of the week) or the Daily frequency: schedule the job to occur once at a specific hour (in the format HH:MM:SS), or the job runs every specified minutes / hours within a given time frame (hours).



7. Click the Save button at the bottom of the window to save the scheduled job.

Step 3. Use indexed values on flows

Once you added all the needed entities to your bot, select solution and on the main menu click Flows. Select a flow and go to the Flow Editor page, click the Training Phrases section and add a complete training phrase (including sample value for the indexed entity to be extracted).

In the training phrase, select the word / sequence of words that is sample value.

Click on the selected word(s). The Entity Extraction pop-up appears.

Click in the Target Entity / Entity field and select the entity that will store the extracted value.

Click the Save button. The pop-up closes and the sample value is highlighted in green and marked with the label Entity.

You can also use the indexed entity on flow steps with input mapping (when you want to do input mapping with NER) and for intent recognition.

After the ML model is trained, you can test a phrase with values for the intended entities (either in Bot Designer / Details or in the Flows section). The DRUID ML engine will identify the correct intent and extract the values from the indexed entity.

Whenever the bot reaches the flow, it extracts the user input and stores it in the entity you provided in the flow utterance (or in the language specific field clone) and validates that value against the list of indexed values. If the value is found, it is available in the conversation flow and you can further use it on the flow based on your needs.